Rear obstruction detection

ABSTRACT

A method is provided using a system mounted in a vehicle. The system includes a rear-viewing camera and a processor attached to the rear-viewing camera. When the driver shifts the vehicle into reverse gear, and while the vehicle is still stationary, image frames from the immediate vicinity behind the vehicle are captured. The immediate vicinity behind the vehicle is in a field of view of the rear-viewing camera. The image frames are processed and thereby the object is detected which if present in the immediate vicinity behind the vehicle would obstruct the motion of the vehicle. The processing is preferably performed in parallel for a plurality of classes of obstructing objects using a single image frame of the image frames.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority under 35 U.S.C. § 119(e) fromU.S. provisional application 60/914,859 filed Apr. 30, 2007, thedisclosure of which is incorporated entirely herein by reference.

FIELD AND BACKGROUND

The present invention relates to vehicle warning systems, and moreparticularly to a system and method for detection of objects behind thevehicle and classification of the objects.

Motor vehicles, especially sport utility vehicles (SUV), pick-up trucksand off-road vehicles typically have a “blind” zone behind the vehicleswhich is a contributing cause for accidents when the vehicles are drivenin reverse. The “blind” zone cannot be viewed directly by the driver norwith the use of the vehicle mirrors. Motor vehicle accidents involvingvehicles being driven in reverse cause vehicle damage there is acollision with typically low obstructions, e.g. walls and poles, andserious accidents involving children.

In an effort to reduce the frequency and severity of vehicular accidentsinvolving vehicles being driven in reverse, vehicle manufacturers haveintroduced rear-view cameras mounted on the vehicle which image the“blind” zone behind the vehicle.

U.S. Pat. No. 7,113,867 disclosed by the present inventor system isdesigned to detect objects in front of the vehicle. A camera is mountedinside a vehicle behind the windshield, and views the environmentthrough the windshield in front of vehicle. A processor analyzes theimages acquired by the camera. The system is operative to detect lanemarkings, pedestrians, other vehicles, obstacles, road signs, and/orother objects up to relatively long distances, typically 50-100 metersfrom the front of the vehicle.

Recognition of obstacles located behind a vehicle present a differentset of image processing issues than addressed by a forward-lookingcamera system. Obstacles located behind a vehicle may be stationary orslowly moving, and at close range, at one to two meters. An additionalfactor to be considered is that vehicles move in reverse under 10km/hour.

Rear looking camera systems which provide the driver additionalvisibility using an attached monitor, for example parking assistsystems, and blind spot detection systems are known in the industry, butunlike the forward looking camera systems, these reverse looking camerasystems provide the driver images on the monitor without providing anyanalysis of the images or warning of the existence of an obstacle behindthe vehicle or of an expected collision with an obstacle.

There is thus a need for, and it would be highly advantageous to have arear viewing camera system mounted in a vehicle which provides analysisfor particularly common classes of obstacles typically found behind avehicle and warnings to the driver based on the analysis when suchobstacles are found.

European Patent EP1876829 discloses a vehicle vicinity monitoring systemwhich captures images of the vicinity of the vehicle using animage-capturing device, and uses a notification device to providevehicle occupants with information concerning obstacles in the vicinityof the vehicle. A primary lighting device and/or a secondary lightingdevice are provided to the vehicle. The primary lighting device directslight upon shadows cast within the imaging field of the image-capturingdevice as a result of the illumination device being illuminated. Thesecondary lighting device is a lighting device for projecting light in aprescribed pattern within the imaging field of the image-capturingdevice in order to confirm the existence of an obstacle.

BRIEF SUMMARY

The term “immediate vicinity” in reference to a vehicle typicallyincludes the ground less than one meter from the vehicle edge in thedirection of intended motion, e.g. reverse motion, or in otherembodiments (e.g. for cameras of field of view 180 degrees or more) theterm “immediate vicinity” induces zero distance from the vehicle edge inthe direction of intended motion in which the camera also views straightdown.

According to an aspect of the present invention, there is provided asystem mounted in a vehicle. A camera is oriented to capture imageframes from the immediate vicinity of the vehicle in a direction ofmotion, e.g. reverse motion, of the vehicle. The immediate vicinity ofthe vehicle including the ground at least one meter from the end of thevehicle in the direction of the motion is in a field of view of thecamera. A processor attached to the camera inputs the image frames. Theprocessor processes the image frames for detection in the immediatevicinity of the vehicle of an object which if present obstructs themotion of the vehicle. A warning mechanism attached to the processorpreferably warns the driver of the vehicle when the object is detected.The detection is typically performed for multiple classes of obstructingobjects using a single image frame while the vehicle is stationary. Theclasses of obstructing objects include spherical balls, vertical poles,hanging chains and upright boxes. Alternatively or in addition, thedetection uses color in a single image frame when the color isdistinguishable from background colors in the vicinity of the vehicle.The detection is preferably performed by performing image differencingbetween the image frames while the vehicle is stationary and while theobject is moving. Typically, a light is attached to the vehicle whichilluminates the immediate vicinity and the detection is performed usingthe image frames by imaging the illumination from the light. When theobject is a wall in the immediate vicinity of the vehicle, a light isattached to the vehicle, illuminating the wall. The detection includesestimating a distance between the vehicle and the wall based on theposition in image space of the illumination from the light. Thedetection preferably uses a meeting point or extrapolation point in oneor more of the image frames between the object and a shadow of theobject. The detection is preferably performed by image differencingbetween one of the image frames without illuminating the immediatevicinity and another one of the image frames while illuminating theimmediate vicinity by the at least one light.

According to another aspect of the present invention a method isprovided using a system mounted in a vehicle. The system includes acamera and a processor attached to the camera. Multiple image frames arecaptured from the immediate vicinity of the vehicle in a direction ofmotion of the vehicle. The immediate vicinity of the vehicle includingthe ground at least one meter from the end of the vehicle in thedirection of the motion is in a field of view of the camera. The imageframes are processed and thereby the object is detected which if presentin the immediate vicinity behind the vehicle would have obstructed themotion of the vehicle. A warning mechanism attached to the processorpreferably warns the driver of the vehicle when the object is detected.The detection is typically performed for multiple classes of obstructingobjects using a single image frame while the vehicle is stationary. Theclasses of obstructing objects include spherical balls, vertical poles,hanging chains and upright boxes. Alternatively or in addition, thedetection uses color in a single image frame when the color isdistinguishable from background colors in the vicinity of the vehicle.The detection is preferably performed by performing image differencingbetween the image frames while the vehicle is stationary and while theobject is moving. Typically, a light is attached to the vehicle whichilluminates the immediate vicinity and the detection is performed usingthe image frames by imaging the illumination from the light. When theobject is a wall in the immediate vicinity of the vehicle, a light isattached to the vehicle, illuminating the wall. The detection includesestimating a distance between the vehicle and the wall based on theposition in image space of the illumination from the light.Alternatively or in addition, the detection preferably uses a meetingpoint or extrapolation point in one or more of the image frames betweenthe object and a shadow of the object.

Alternatively or in addition, a first image frame is preferably capturedwithout illuminating the immediate vicinity by the light; and whilingilluminating the immediate vicinity by the light a second image frame iscaptured. The processing is performed by image differencing between thefirst image frame and the second image frame.

According to yet another aspect of the present invention a method isprovided using a system mounted in a vehicle. The system includes arear-viewing camera and a processor attached to the rear-viewing camera.When the driver shifts the vehicle into reverse gear, or when the driverreleases the brakes prior to motion in reverse and while the vehicle isstill stationary or moving slowly, image frames from the immediatevicinity behind the vehicle are captured. The immediate vicinity behindthe vehicle is in a field of view of the rear-viewing camera. The imageframes are processed and thereby the object is detected which if presentin the immediate vicinity behind the vehicle would obstruct the motionof the vehicle. The processing is preferably performed in parallel andoptionally simultaneously for a plurality of classes of obstructingobjects using a single image frame of the image frames. When the drivershifts the vehicle into reverse gear, the reverse lights of the vehicleare turned on. Image differencing is preferably performed between imageframes captured before and after the reverse lights are turned on.Similarly, image differencing may be performed before and after the rearbrakes lights are turned off when the driver releases the brakes beforeinitiating the reverse motion of the vehicle.

The foregoing and/or other aspects will become apparent from thefollowing detailed description when considered in conjunction with theaccompanying drawing figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, withreference to the accompanying drawings, wherein:

FIG. 1 illustrates a simplified exemplary diagram of a system, accordingto an embodiment of the present invention.

FIG. 1A illustrates schematically inputs and outputs of the system ofFIG. 1, according to an aspect of the present invention;

FIG. 2 illustrates a chain hanging between posts as an example of anobject in a class of obstructions, detectable according to features ofthe present invention;

FIG. 3A and FIG. 3B illustrate aspects of the present invention, showingimages captured from a rear viewing camera after the vehicle is shiftedinto reverse gear;

FIG. 4 is a graphical representation showing features of the image ofFIG. 3A;

FIGS. 5A, 5B illustrate two images of an outdoor scene, with and withoutrear-light illumination, according to an aspect of the presentinvention;

FIG. 5C illustrates an image difference between the images of FIG. 5Aand FIG. 5B;

FIG. 5D illustrates thresholding and difference imaging between theimages of FIG. 5A and FIG. 5B;

FIG. 6 illustrates an image after processing with a combination of imagedifferencing and Canny edge detection, according to an embodiment of thepresent invention; and

FIG. 7 illustrates a reverse light integrated with a lens array,according to a feature of the present invention.

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments of the presentinvention, examples of which are illustrated in the accompanyingdrawings, wherein like reference numerals refer to the like elementsthroughout. The embodiments are described below to explain the presentinvention by referring to the figures.

Before explaining embodiments of the invention in detail, it is to beunderstood that the invention is not limited in its application to thedetails of design and the arrangement of the components set forth in thefollowing description or illustrated in the drawings. The invention iscapable of other embodiments or of being practiced or carried out invarious ways. Also, it is to be understood that the phraseology andterminology employed herein is for the purpose of description and shouldnot be regarded as limiting.

While the discussion herein is directed toward application of thepresent invention to a rear viewing system mounted on a vehicle, e.g.automobile, in order to provide warnings to the driver while the vehicleis moving in reverse, the present invention may, by non-limitingexample, alternatively be configured as well using side motion or frontmotion for special vehicles, e.g. trucks, cranes, forklifts, tanks,robots, other than typical automobiles.

Referring now to the drawings, FIG. 1 shows a simplified diagram of asystem 10, according to an embodiment of the present invention. Arear-view system 100 installed in vehicle 10, having a camera 136 and aprocessor 132, is situated above ground plane 20. System 100 detects andpreferably classifies objects situated, in the immediate up to two tothree meters behind vehicle 10. System 100 includes one or more lights12, which in different aspects of the present invention may be one ormore of taillights, brake lights, reversing lights and turn indicators.

Each object 138 detected at the rear of vehicle 10, is assumed to bedisposed on ground plane 20. Hence, a distance Z to each detected object138 is measured from the bottom of detected object 138 to camera 136.Distance Z from vehicle 10 to detected object 138 is approximatelymeasured by processor 132 using the following equation:

$\begin{matrix}{Z = {{- f}\frac{H}{y}}} & (1)\end{matrix}$

wherein H is the physical height of camera 136 with respect to groundplane 20 and y is the vertical displacement in image space of the bottomof the detected object. Aspects of measuring distance to a detectedobject, using a camera mounted on a vehicle, are disclosed by Stein etal. in US patent publication No. 20070154068 the disclosure of which isincorporated by reference as if fully set forth herein.

Camera 132 is preferably configured to have a wide field of view, e.g.130-180 deg along at least one axis. The wide field of view ispreferably oriented, e.g. horizontally to capture images from behind theentire width of vehicle 10. The wide field of view in at least one axismay be achieved using an rectangular image sensor, an anamorphic opticaldesign and/or using a cylindrical lens design as is known in the art ofoptical design. Standard wide angle lenses are achieved using fish eyelenses or aspheric designs.

Camera 132 is typically mounted less than one meter above ground plane20, preferably just above or below the license plate of vehicle 10 andtilted downward in order to focus on ground plane 20. The minimum depthof field of camera 132 is preferably short at about 50 centimeters.Typically, one or more camera parameters e.g. aperture, exposure timeare adjusted according to lighting conditions, especially betweendaytime and nighttime. The camera preferably includes an adjustablefocus which may be used to focus on obstacles (optionally) in real timeat 1-2 meters range.

A commercial camera lens suitable for camera 132 is Sunex Model DSL215(Sunex Inc., Carlsbad, Calif., USA) featuring a 185° field of view witha relative aperture of F/2.0 and an integrated infrared (IR) cut-offfilter. A suitable CMOS image sensor is Firefly® MV available fromPointgrey Research, Vancouver, Canada.

Although in FIG. 1, object 138 is shown as a ball, object 138 usingdifferent features of the present invention may be in other classes ofobjects including balls, toys, walls, vertical poles, chains, ropes andpersons especially small children which may be found as obstacles behindvehicle 10.

Reference is now made to FIG. 1A which illustrates schematically inputsand outputs of rear-view system 100, according to different embodimentsof the present invention. Inputs to system 100 include a video input 145from rear view camera 136. Other input signals 141 preferably includeone or more signals such as speed, steering, brake pedal activation,brake light activation, gear position, and rear light activation. Inputsignals 141 are typically available to system 100 from the CAN bus 149of vehicle 10. Controller Area Network (CAN) is a computer networkprotocol and bus standard designed for automotive applications whichallows microcontrollers and other devices to communicate with each otherand without a host computer. Outputs 143 of system 100 are objects 138or potential obstructions either in image space and/or in threedimensional physical space behind the car. Outputs 143 are preferablyprovided by specifying range Z and angle. Outputs 143 are preferablyused be used to highlight information on a video monitor 147, giveaudible or other visible warnings, e.g. on monitor 147, to the driverand/or activate safety systems such as brakes of vehicle 10.

At the core of system 100 is image processing module 132 whichincorporates a variety of vision algorithms each suitable for adifferent class of object, e.g. a general object of undefined shape,pole, pedestrian, ball. The algorithms used typically vary based on thestate of vehicle 10, e.g. stationary or moving. According to a featureof the present invention, algorithms are initiated for obstructiondetection just before vehicle 10 starts to move backwards since oncevehicle 10 is in motion there is less time to warn the driver regardingthe obstruction. When vehicle 10 is stationary, image differencingbetween frames can be used to indicate a potential obstruction andprovide a warning to the driver. The image differencing preferablyutilizes the change in illumination in the vicinity of the vehicle whenthe brake lights are turned off just prior to moving the vehicle, and/orthe reverse lights being turned on when the vehicle is shifted intoreverse gear. According to different aspects of the present invention,obstruction detection includes detection of pedestrians particularlysmall children, balls, detection using color, detection using extendedvertical lines, detection of vertical poles, detection of hangingchains, and detection of upright rectangles or boxes.

Pedestrian Detection

Pedestrian detection and classification is described in detail in thefollowing reference for vehicle 10 being either moving or stationary inUS patent application 20070230792, and in A. Shashua, Y. Gdalyahu and G.Hayon, “Pedestrian Detection for Driving Assistance Systems: SingleframeClassification and System Level Performance”, in Proc. of the IEEEIntelligent Vehicles Symposium (IV2004), June 2004, Parma, Italy.

Detection from a Stationary Vehicle Using a Single Frame

Detection of Balls:

Spherical objects are a particularly important class of objects as theyare often children's toys. If a ball is present behind vehicle 10, achild might be close by and might decide to try and save their ‘prized’possession. Thus, a feature of the present invention includes detectingspherical objects 138. Spheres 138 are preferably detected by searchingfor circles in the image using a Hough transform on the image after edgedetection has been performed within the image. Known algorithms for edgedetection as developed for instance by John F Canny including thefollowing steps: noise reduction, intensity gradient of the image,non-maximum suppression of the intensity gradient, tracing edges throughthe image with hysteresis thresholding and/or differential edgedetection. Any circle found in an image frame is assumed to represent aball lying on the ground. The distance to the ball (object 138) is givenby applying equation (1) based on the ground plane 20 constraint. Thebottom of the circle in the image is assumed to be a point on groundplane 20. According to a feature of the present invention a warning issounded whenever a driver shifts to reverse and a sphere is detectedwithin a certain range Z. Balls which are located above the ground willgive a range too large but these are less critical as they are morelikely to be balls on the shelf or sidewalk and thus not signifying thesame danger. A circle detection which is below ground plane 20 may alsoresult in a warning to the driver but a hole is also a sufficientobstruction worth providing a warning. Since most planar circles in thethree dimensional world are either lying flat on the ground, e.g.manhole cover, or on vertical planes (e.g. traffic signs, drawings onwalls etc), these circles typically appear as ellipses in the image andthus will not cause many false warnings. A traffic sign right behind thevehicle would appear as a circle. A traffic sign recognition algorithmis optionally performed and once a traffic sign is confirmed the twoparallel lines protruding below the circular sign are taken to be thepole.

Detection Using Color

Particular colors (such as red, orange and blue) are typical forobstacles and not for clear road surfaces. By searching in the HSV(hue,saturation,value) color space, image regions are detected withthese hues above a threshold value of saturation. These colored imageregions can be used as inputs for obstruction detection algorithms,according to other features of the present invention. For example, thebottom edge of a colored ball in the image can be used to estimate thedistance to the colored ball using the ground constraint equation (1). Awarning can be given when a colored ball is close and in path of therearward moving vehicle 10 if the driver shifts into reverse.

Detection of Obstacles Using Extended Lines

A vertical line is an image that extends across the horizon line in animage of an object of interest, i.e. a potential obstruction 138 sincethe object is clearly not a lane or road marking when object 138 extendsabove road plane 20 and is not at infinite distance. The bottom point inthe vertical line in the image (on ground plane 20) gives an upper boundto the range between vehicle 10 and object 138 and can be used fordirectly warning the driver. Vertical image lines when detected can beused to trigger attention to other detection modules, according to otherfeatures of the present invention for instance the detection of verticalpoles.

Detection of Vertical Poles

Pairs of substantially vertical extended lines detected close togetherin an image indicate that object 138 is a vertical pole. If the pole isdirectly inserted into the ground the ground plane constraint, equation(1), gives an accurate estimate of the range Z. If the pole is on araised base then the distance estimate is too large. It is thereforeuseful to correctly detect the base of the pole. According to a featureof the present invention, we look for circular and/or square bases. Acircular base appears in the image as an ellipse. However, not allellipses are valid since the circle (cross section of the cylindricalbase) is in a plane parallel to ground plane 20 and is centered aroundthe pole. Since the difference due to the ellipse height above groundplane 20 is small we can search in image space for ellipses which areimages of circles on the ground plane and parameterized by distance andradius. The 3rd degree of freedom of circles on the ground plane isconstrained since the circle center must lie on a line equidistant fromthe two lines defining the pole.

Detection of Hanging Chains

It is often the case that parking spaces are closed by chains hangingbetween two posts. According to a feature of the present invention, thedriver is warned before backing up into such a chain. It is well knownthat a hanging chain forms a catenary curve,defined by the followingequations:

$\begin{matrix}{{y = {\overset{\prime}{a}\left( {{\cosh \; \left( \frac{x}{\overset{\prime}{a}} \right)} - 1} \right)}},} & (2)\end{matrix}$

wherein

$\begin{matrix}{{\overset{\prime}{a} = \frac{T_{0}}{\overset{¨}{e}}},} & (3)\end{matrix}$

Reference is now made to FIG. 2, which illustrates a chain 60 hangingbetween posts 62 in the immediate vicinity of the rear of vehicle 10.Chains 60 that lie in the plane perpendicular to the rearward directionof vehicle 10 are typically considered. Furthermore we are looking forchains 60 within a certain length range (e.g. 1 to 4 meters) and with acertain degree of sagging (e.g. 2% to 10% of the curve length). Chain 60appears in the image as a perspective transformation of this curve. Analgorithm for detecting chain 60, according to a feature of the presentinvention is as follows:

1. Run an edge detection algorithm on the image from camera 136 tolocate long lines.

2. Select lines that are not straight, have a local minimum, e.g. point64, and curve upwards.

3. Each local minimum is a candidate minimum of a catenary curve. Thisleaves two degrees of freedom: (i) the distance to the minimum point(which given the known camera geometry gives the three dimensionalphysical coordinate of the point) and (ii) the curve parameter_(alpha)from equation (2).

4. Search the two dimensional image space for the best match with pointson the line. The result from this stage is a curve segment that extendsfrom the candidate minimum.

5. For each possible distance of the chain to vehicle 10 compute thethree dimensional physical location of the end points of chain 60 andminimum point 64. Discard lines that do not meet the length andcurvature criteria as defined above.

6. Once a best fit has been found, check to see if there are verticallines or poles (pairs of vertical extended lines) near the ends. Theseare candidate posts 62 which support chain 60. The bottom of thesevertical lines is assumed to be on ground plane 20 and thus gives adistance estimate to chain 60 from vehicle 10 using equation (1).

Detection of Upright Rectangles and Boxes

Rectangular surfaces which are perpendicular to ground plane 20 are ofinterest. In particular these could be the side of a box. To detectrectangular surfaces, we look in image space for two vertical linesjoined at the top and bottom by two straight lines. Due to perspective,these top and bottom lines need not be parallel. The bottom line isassumed to lie on ground plane 20 and thus gives a distance to the twovertical lines. This in turn gives the three dimensional physicalcoordinates of the upper line. In particular we can check if the upperline appears to be parallel to ground plane 20.

If the upper line appears to be parallel to the ground plane then theimage region is a candidate upright rectangle. To be considered anobstruction, rectangles that are in the vehicle path are kept which arewithin a certain distance and not too large. For example, the objectmust be lower than a certain minimal height, e.g. 1 meter, and at leastone of the upright lines must be within the vehicle path. Anythinglarger than the minimal height is typically noticed by the driver. Anextra check can be performed such as checking for possible transparency.Rectangles are typically rejected in which at least one of the lines iscut by another line in the image.

Detection from a Stationary Vehicle Using Multiple Frames and ObjectMotion

When vehicle 10 is stationary, image differencing can be used betweenimage frames to detect moving objects. In particular, image differencingbetween frames is used for detecting obstructions 138, e.g. balls,people and other vehicles which suddenly enter the field of view ofcamera 136. An algorithm for detection based on image differencing,according to aspects of the present invention is as follows:

1. First, simple image differencing between multiple image frames isperformed to produce a difference image. The intensity (and/or color) ofcorresponding picture elements (pixels) of two image frames from camera136 are compared. The difference image is derived from the comparison,typically the pixel-by-pixel intensity/color difference between the twoimage frames.

2. The difference image is thresholded and portions of the differenceimage with intensity greater than a threshold are preferably connectedto form connected components patches.

3. For each connected component patch, the optical flow is preferablycomputed. We are looking for a more or less uniform optical flowdirection over the patch. Optical flow is the pattern of apparent motionof objects, surfaces, and edges in a visual scene caused by the relativemotion between an observer (an eye or a camera) and the scene.

4. If the dominant optical flow component is measured over three or moreframes to see if it is consistent. These last two steps help removefalse alerts due to moving shadows from overhead trees.

5. The bottom of the connected component is assumed to be on groundplane 20 and this gives a distance estimate using equation (1).

6. The object detected is tracked over time, i.e. multiple image frames.If the object then stops moving, the final position is stored in storageattached to processor 132

7. A warning is preferably given if the driver shifts into reverse gearand an object was detected based on the image difference in the path ofvehicle 10 within a certain range Z.

Detection from a Stationary Vehicle Using Multiple Frames and RearLights

Referring back to FIG. 1, vehicle 10 typically has four types of lights12 at the back: taillights, brake lights, reversing lights and turnindicators. When a driver intends to drive vehicle 10 in reverse, thedriver typically presses on the brakes thus turning on the brake lights,and then the driver shifts into reverse thus turning on the reverselights. The location of lights 12 relative to the camera is known andthe exact timing of switching on of lights 12 can be provided to system100 (e.g. from the CAN bus). According to features of the presentinvention, change analysis, e.g. color/intensity image differencing isperformed between two image frames from camera 136, using one imageframe just before the exact moment lights 12 go on (either the whitereverse lights and/or the red brake lights) and another image frame justafter lights 12 go on. Using the image differencing, changes in thephysical scene as viewed by camera 136 due to turning on vehicle lights12 can be determined. This image information can be used in multipleways, according to different aspects of the present invention and andcan provide important information about the physical three dimensionalarea behind vehicle 10 and the presence of obstacles 138.

In order to deal with the lights 12 being switched on exactly whenparameters (e.g. gain, exposure) of camera 132 are changed a proceduremay be followed according to a feature of the present invention. Assumecamera 132 is going from setting 1 to setting 2 due to changes inlighting, for instance, the sun appears from behind a cloud. Firstcamera 132 is switched from a first setting to a second setting, andthen switched back to the first setting for at least one frame.Afterward the at least one frame, the second setting is restored. Inthis way any event of switching on lights 12, the switching is straddledby two frames with the first setting or two frames with the secondsetting or both.

Estimating the Distance to a Wall

Reference is now made to FIGS. 3A and 3B, which illustrate aspects ofthe present invention, showing images from camera 136 after vehicle 10is shifted into reverse gear. In FIG. 3B vehicle 10 is closer to theback wall than in FIG. 3A. Bright patches 92A, 92B of light on the backwall are from reverse lights 12. When vehicle 10 is closer to the wallin FIG. 3B, each of reverse lights 12 appears farther from the center ofthe image.

The geometry of lights 12 relative to camera 136 is fixed. So, forexample, in the two images shown in FIGS. 3A and 3B the relativelocation of the two bright patches 92 on the wall gives the distance tothe wall, according to an aspect of the present invention.

The following algorithm, according to a feature of the presentinvention, for determining distance to a wall, assumes typical rearlights 12 which are aimed directly rearwards. Lights 12 in this examplehave no special illumination characteristics and are assumed toilluminate with a cone of light which is brightest in the center of thecone. These cones of light appear as bright spots 92 on the wall behindvehicle 10.

1. Detect switch on rear lights from the CAN bus or perform directanalog to digital conversion of the voltage to the lights.

2. Subtract image from before lights were turned from the image afterthe lights ware turned on.

3. Detect centers (or centroids) of one or both bright patchescorresponding to one or both rear lights.

4. Compute distance to wall based on the following equation:

$\begin{matrix}{Z = {{- f}\frac{X}{x}}} & (4)\end{matrix}$

where f is the camera focal length, X is the lateral position of therear light relative to the camera and x, the position of the peakbrightness of the spot. The location of the centers/centroids of spots92 does not necessarily need to be accurately determined since thedistance X is large (0.6 m to 0.9 m) and the relevant distance to thewall is short (1-2 meters). With such parameters and error in centerposition of 10 pixels gives a wall distance error on the order of 20 cm.which is acceptable for most applications

Fine tuning of the distance can be performed by searching the bottom ofthe image for horizontal lines which are within the error bounds, _, ofthe distance found using reverse lights 12 assuming they lie on theground plane.

$\begin{matrix}{\left( {{f\frac{H}{y}} - Z} \right) < \overset{{^\circ}}{a}} & (5)\end{matrix}$

where H is the camera height, y is the vertical position of the line inthe image Z is the distance computed using reverse light spots 92.

It is possible to improve the algorithm if the lighting pattern 92 fromreverse lights 12 has a more complex illumination pattern for instancewith strong edges or center. This is often the case in practice andfurthermore the reverse lights internal mirror surfaces and externalplastic lens case may be designed as is known in the art of opticaldesign to illuminate with a a more complex illumination pattern. Forexample, if the reverse light produces a somewhat vertical streak oflight it is possible to more accurately estimate distance along thevertical streak.

Equation (4) is relevant for light beams parallel to the optical axis ofvehicle which is the typical case for light-emitting units such as rearlights which are configured to shine straight back. However, if thebeams are not parallel, the angle of the beams is known a priori or bycalibration, the distance to the wall can be calculated using standardmethods of “stereo vision”, as is known in the industry. Theintersection of a light beam and the line of sight of the camera pixelprovides the three dimensional physical location of the intersection ofthe light beam and the obstacle. Reference is now made to FIG. 7 whichillustrates a desired specification of light source, e.g. reverse light12 designed with a lens array 71 integrated into the cover of lamp 12,according to a feature of the present invention. Lens array 71preferably focuses in spots at distances between one and two meterswhile providing uniform illumination of longer distances. The advantagelens array 71 is that they concentrate the light at or near the focaldistance of camera 132 so that easier to image the focal spots of light12 during daylight hours.

Obstacle Detection Using Shadows on the Ground Plane

The images of FIG. 3A, 3B show an example of shadows created by reverselights 12 (or brake lights) due to physical obstacles, e.g. cinderblocks 80L and 80R shown on the floor of the images of FIGS. 3A and 3B.

Reference is now also made to FIG. 4 which is a graphical representationof the image of FIG. 3A. Ground plane 20 is shown and rear wall 90.Bright spots 92L and 92R from lights 12 are shown. Centroid positions94L and 94R within bright spots 92L and 92R are shown respectively.Since the relative position of the light source, e.g. reverse light 12and camera 136 is precisely known and the location and angularorientation of camera 136 relative to ground plane 20 is also known,possible matches may be found between imaged edges Ei of objects, e.g.cinder blocks 80L and 80R in the image and shadow edges Sj in the image.Once matches are found these can be used to determine the threedimensional position of objects, e.g. cinder blocks 80L and 80R.

An exemplary algorithm follows for detecting vertical obstacles, e.g.cinder blocks 80L and 80R, according to an aspect of the presentinvention:

1. Detect preferably all straight vertical lines in the image. Verticalstraight lines in the image are potentially from edges Ei of threedimensional physical obstacles e.g. cinder blocks 80L and 80R.

2. Detect preferably all straight edges in the image which appear afterlights 12 are switched on. These are potentially edges Sj of shadows ofobstacles

3. Test preferably each pair (Ei, Sj) to see if it is possible to matchwith tests as follows:

(a) Pick lines that meet at a point or that their extensions meet at apoint; lines that cross are typically rejected.

(b) The angle between lines (Ei, Sj) must be less that 90 degrees.

(c) The point of meeting must be at the bottom of the vertical line (orits extension).

4. Test pairs (Ei, Sj) that passed the tests (a), (b), (c) above usingthe precise geometry of the camera and light source.

(a) Assume the meeting point in the image of the vertical line and theshadow line (x, y), is on the ground plane. Compute the location of thepoint in three dimensional world coordinates:

$\begin{matrix}{Z = {f\frac{H}{y}}} & (6) \\{X = {f\frac{Z}{x}}} & (7)\end{matrix}$

where Z is the longitudinal distance from the camera and X the lateraldistance.

(b) Project the expected shadow location Se of the vertical line giventhe location of light source 12 and the location of camera 136.

(c) Compute maximum image distance between points on the candidateshadow line Sj and the computed one Se. If this distance is below athreshold then the match is preferably accepted.

Although the above algorithm assumes the vertical lines are from sharpedges, a similar algorithm may be applied to vertical cylinders. It isalso possible to try to match curved lines in the image that come fromupright objects in the world (such as pedestrians or poles) to curvedlines that belong to their corresponding shadows. For a curved line S wecan hypothesize that lines S comes from such an upright object and then‘guess’ a distance. For each point along the curve S one can compute theexpected location of the shadow given that distance. One can then see ifcurve S aligns with an edge point in the illuminated image (which mightbe a candidate shadow edge point). For each ‘guessed’ distance one cancount the number of points that match a possible shadow edge point andpick the distance that matches the largest number of points. If certainpercentage of matched points is above a threshold a matched curve atthat distance is detected.

Detection of General Obstacles Using Brightness Changes

In bright outdoor scenes it might not be possible to see actual shadowson the ground due to rear lights. However, since the angle ofillumination from rear-lights 12 is very different than from theoverhead sun, some change in illumination may be detected on objects inthe scene. Reference is now made to FIG. 5. FIGS. 5A and 5B illustratetwo images of an outdoor scene, with and without rear-lightillumination. When image differencing is applied (FIG. 5C bottom left)and thresholding difference imaging (FIG. 5D bottom right) a brightnesschange is clearly seen on the close upright object which in this case isa pedestrian located about one meter behind vehicle 10. The solidhorizontal line in FIG. 5B highlights a line in the image that projectsto points on the ground approximately 2.5 m behind the camera. Giventhat there is considerable ambient lighting during daylight, anoticeable change in illumination on points at that distance and onsurfaces parallel to the ground is not expected. The fact that a changein illumination is detected indicates we have an object quite a bitcloser.

The bottom of the bright blob in difference images (FIGS. 5C, 5D) givesan estimate of the distance to the pedestrian. In this case the resultwas 1.3 m. Fine tuning of the distance estimates can be done bycombination with edge images. Reference is now made to FIG. 6 whichillustrates an image from system 100 after processing with a combinationof image differencing and Canny edge detection, according to an aspectof the present invention. With edge detection, the extent of the visibleblob may be extended downward toward ground plane 20, estimated by apoint with no detectable edges. The distance to the pedestrian may thenbe estimated using the ground constraint equation (1). A warning can begiven to the driver if a suspected object is detected close behind thevehicle so that the drive. In embodiments of the present invention inwhich a monitor is connected to system 100, the bright blob which issuspected to be an obstacle is preferably highlighted.

While the invention has been described with respect to a limited numberof embodiments, it will be appreciated that many variations,modifications and other applications of the invention may be made.

1. A system mounted in a vehicle, the system comprising: (a) a cameraoriented to capture a plurality of image frames from the immediatevicinity of the vehicle in a direction of motion of the vehicle, whereinthe immediate vicinity of the vehicle including the ground at least onemeter from the end of the vehicle in the direction of the motion is in afield of view of said camera; and (b) a processor attached to saidcamera inputs the image frames and processes the image frames fordetection in the immediate vicinity of the vehicle of at least oneobject which if present obstructs the motion of the vehicle.
 2. Thesystem, according to claim 1, further comprising: (c) a warningmechanism attached to said processor, wherein said warning mechanismprovides a warning to a driver of the vehicle when said at least oneobject is detected.
 3. The system, according to claim 1, wherein saiddetection is performed for a plurality of classes of obstructing objectsusing a single image frame of said image frames while the vehicle isstationary.
 4. The system, according to claim 3, wherein said classesare selected from the group consisting of: spherical balls, verticalpoles, hanging chains and upright boxes.
 5. The system, according toclaim 3, wherein said detection uses color in a single image frame ofsaid image frames, whereby said color is distinguishable from backgroundcolors in the vicinity of the vehicle.
 6. The system, according to claim1, wherein said detection is performed by performing image differencingbetween said image frames while the vehicle is stationary and while saidat least one object is moving.
 7. The system, according to claim 1,further comprising: (d) at least one light attached to the vehicle,wherein said at least one light illuminates said immediate vicinity,wherein said detection is performed using said image frames by imagingillumination from said at least one light.
 8. The system, according toclaim 1, wherein said at least one object is a wall in the immediatevicinity of the vehicle, the system further comprising: (d) a lightattached to the vehicle, wherein said light illuminates said wall,wherein said detection includes estimating a distance between thevehicle and said wall based on the position in image space of theilluminations from said light.
 9. The system, according to claim 1,further comprising: (d) at least one light attached to the vehicle,wherein said at least one light illuminates said immediate vicinity,wherein said detection uses a meeting point or an extrapolation point inat least one of said image frames between said at least one object and ashadow of said at least one object.
 10. The system, according to claim1, further comprising: (d) at least one light attached to the vehicle,wherein said at least one light is capable of illuminating saidimmediate vicinity, wherein said detection is performed by imagedifferencing between one of said image frames without illuminating saidimmediate vicinity by said at least one light and another one of saidimage frames while illuminating said immediate vicinity by said at leastone light.
 11. In a system mounted in a vehicle, the system including acamera and a processor attached to said camera, the method comprisingthe steps of: (a) capturing a plurality of image frames from theimmediate vicinity of the vehicle in a direction of motion of thevehicle, wherein the immediate vicinity of the vehicle including theground at least one meter from the end of the vehicle in the directionof the motion is in a field of view of the camera; and (b) processingthe image frames thereby detecting in the immediate vicinity of thevehicle at least one object which if present obstructs the motion of thevehicle.
 12. The method according to claim 11, further comprising thestep of: (c) upon said detecting, warning a driver of the vehicle of apotential obstruction.
 13. The method according to claim 11, whereinsaid processing is performed in parallel for a plurality of classes ofobstructing objects using a single image frame of said image frameswhile the vehicle is stationary.
 14. The method according to claim 11,wherein said processing includes image differencing between said imageframes while the vehicle is stationary and while said at least oneobject is moving.
 15. The method according to claim 11, furthercomprising the step of: (c) illuminating said immediate vicinity by atleast one light attached to the vehicle, wherein said processing isperformed based on said illuminating.
 16. The method according to claim11, wherein said at least one object is a wall in the immediate vicinityof the vehicle, the method further comprising the step of: (c)illuminating said wall by a light attached to the vehicle, wherein saidprocessing includes estimating a distance between the vehicle and saidwall based on the position in image space of the illumination from saidlight.
 17. The method according to claim 11, the method furthercomprising the step of: (d) illuminating said immediate vicinity by atleast one light attached to the vehicle, wherein said processing uses ameeting point or extrapolation point in at least one of said imageframes between said at least one object and a shadow of said at leastone object.
 18. The method according to claim 11, wherein at least onelight is attached to the vehicle, the method further comprising thesteps of: (d) capturing a first image frame of said image frames withoutilluminating said immediate vicinity by said at least one light; and (e)while illuminating said immediate vicinity by said at least one lightcapturing a second image frame of said images; wherein said processingis performed by image differencing between said first image frame andsaid second image frame.
 19. In a system mounted in a vehicle, thesystem including a rear-viewing camera and a processor attached to saidrear-viewing camera, the method comprising the steps of: (a) whileshifting the vehicle into reverse gear, capturing a plurality of imageframes from the immediate vicinity behind the vehicle, wherein theimmediate vicinity behind the vehicle is in a field of view of therear-viewing camera; and (b) processing the image frames therebydetecting in the immediate vicinity behind the vehicle at least oneobject which if present obstructs the motion of the vehicle.
 20. Themethod according to claim 19, wherein said processing is performed inparallel for a plurality of classes of obstructing objects using asingle image frame of said image frames.
 21. (canceled)